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Background / Context: 

In this presentation we will present about an ongoing partnership with the Philadelphia 
School District (PSD) to implement and research the Ongoing Assessment Project (OGAP). 
OGAP is a systematic, intentional and iterative formative assessment system grounded in the 
research on how students leam mathematics. The OGAP system consists of an integrated into a 
set of tools, practices, support materials, and in-depth professional development. OGAP draws 
on two overlapping lines of research that have informed recent efforts to improve classroom 
instruction. First, teachers’ skilled use of ongoing formative assessment to tailor instruction to 
student needs is a key factor in improving student learning in mathematics (Black & Wiliam, 
1998). The second line of research focuses on the development and use of learning trajectories to 
specify student development within specific domains (Daro et ah, 2011; Sztajn et ah, 2012). A 
learning trajectory orientation can enhance the formative assessment process by providing 
teachers with a clear articulation of learning goals, a framework for how student thinking 
develops, and activities that are likely to move students along the path toward achieving those 
goals (Daro et al., 2011; Heritage, 2008). 

OGAP was funded by the National Science Foundation in 2014. Initially funded for New 
York City, the project site was switched to Philadelphia due to the NYC political context at the 
time of school selection. The project team negotiated access to Philadelphia and worked with the 
SDP leaders to develop the sampling frame for the study and recruit participants. This included 
attending regional leadership team meetings and visiting schools to describe the study to faculty. 
The project team also coordinates data collection, including primary and administrative data, 
with SDP leaders. 

Research Design: 

The program is being implemented and studied as a mixed method randomized control 
trial of OGAP impacts in grades 3-5 in Philadelphia area schools. In the spring of 2014, CPRE 
recruited 60 schools (including charters) in the Philadelphia area to participate in the research 
study of OGAP. These schools were randomly assigned to treatment and control groups, 
stratified by charter school and district. 

Focus of Study: 

The formal research study of OGAP is designed to address the following five central 
research questions, which are broken down into three implementation questions and two impact 
questions: 

Implementation questions include: (1) How do teachers understand the OGAP formative 
assessment process and use it in their daily practice? (2) Is the frequency and sophistication of 
teachers’ use of OGAP associated with greater increases in mathematical knowledge for teaching 
or ability to interpret evidence in student work? And (3) Is the frequency and sophistication of 
teachers’ use of OGAP associated with greater student learning? Impact questions are as 
follows: (4) Does teachers’ use of OGAP increase mathematical knowledge for teaching and 
ability to interpret evidence in student work? What moderating factors affect this relationship? 
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And (5) Does teachers’ use of OGAP significantly improve student perfonnance on both 
proximal and distal student outcome measures? What mediating and moderating factors affect 
this relationship? 

In order to address these research questions, the research team has had to develop a solid 
relationship with both district and school personnel. A key aspect of this is working with district 
personnel to facilitate the collection of implementation data on an externally implemented 
program. 

Instruments: 

The OGAP evaluation features a core set of implementation and teacher and student 
impact measures which are described below. In addition, the study contains a set of qualitative 
data collection and analyses plans which are not contained in this description. 

Teacher Impact Measures. 

Measure of Mathematical Knowledge for Teaching (MKT). Drawing on practice-based 
theory of content knowledge (Shulman, 1987), researchers at the University of Michigan 
developed multiple-choice items that measure the mathematics used in teaching (Hill et ah, 

2008; Ball, Thames, & Phelps, 2008). 

Teacher Analysis of Student Knowledge (TASK). The TASK is an authentic, 
contextualized measure of teachers’ ability to analyze students’ mathematical thinking within a 
grade-specific content area in relation to research-based learning progressions, and formulate 
reasoned instructional responses. Each on-line TASK presents a teacher with a carefully 
designed grade-appropriate set of student responses to a mathematics problem. The TASK is 
well aligned with OGAP in that it measures three key domains of the formative assessment 
process related to the specific mathematical concept that is being assessed: 1) Teachers’ analysis 
of student understanding, 2) Teachers’ knowledge of mathematical learning, and 3) Teachers’ 
informed instructional decision-making. (Supovitz & Sirinides, 2012; Ebby & Sirindes, 2015). 
Teacher Implementation Measure 

Logs of teacher practice. At four randomly assigned intervals over the course of the 
second year of the intervention, teachers are asked to complete a short on-line log that will 
measure their frequency of use of OGAP items, what they do with the resulting student work, 
and how they apply it to their instruction. These logs will be converted into an implementation 
scale to estimate both the frequency and sophistication of teacher use of OGAP. 

Student Impact Measures 

We use two approaches to measure individual-level Grades 3-5 student outcomes. First, 
as a proximal measure to OGAP, we will administer a half-hour test of open-ended items to 
grade three through five students in treatment and control schools at the beginning and end of the 
first and second years of project implementation. The grade-specific assessments will be 
composed of items that ask students to solve problems and show their work (to allow for analysis 
of both performance and strategy sophistication). The assessments will be constructed from 
items pulled from the OGAP item bank (and removed from the intervention). 

Presentation Focus 

The presentation will include a brief description of the study and year one experimental 
impacts on teachers and students. The main emphasis of the presentation will be on the 
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development of the partnership and collaborative work conducted with the SDP, which has 
resulted in a follow-up proposal for deeper work. The presentation will be organized as an 
external research partner’s perspective about working with the district in the following areas: (a) 
facilitating access to and collecting data from schools; (b) organizing the delivery of professional 
development amidst a variety of district initiatives and priorities; (c) providing information and 
feedback to the district to keep them aware of progress and challenges; (d) working 
collaboratively to make sense of early impacts, and; (e) developing ways to deepen the work. 
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Appendices 
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